Virtual microscopy is a technology that gives physicians the ability to navigate and observe a biological specimen at different simulated magnifications and through different 3D views as though they were controlling a microscope. This can be achieved using a display device such as a computer monitor or tablet with access to a database of microscope images of the specimen. There are a number of advantages of virtual microscopy over traditional microscopy: the specimen itself is not required at the time of viewing, thus facilitating archiving, telemedicine and education. Virtual microscopy can also enable the processing of the specimen images to change the depth of field and to reveal pathological features that would be otherwise difficult to observe by eye, for example as part of a computer aided diagnosis system.
Capture of images for virtual microscopy is generally performed using a high throughput slide scanner. The specimen is loaded mechanically onto a stage and moved under the microscope objective so that images of different parts of the specimen can be captured by a sensor. Adjacent images typically have an overlap region so that the multiple images of the same specimen can be combined into a 3D volume in a computer system attached to the microscope. If the specimen movement could be controlled accurately enough these images could be combined directly to give a seamless 3D view without any defects. Typically however this is not the case and the specimen movement introduces errors in position and rotation of the neighbouring images. In this case software algorithms are required to register both the neighbouring images at the same depth and at different depths to reduce defects and distortion between adjoining images.
Microscopy is different from other mosaicking tasks in a number of important ways. Firstly, the specimen is typically moved by the stage under the optics, rather than the optics being moved to capture different parts of the subject as would take place in a panorama capture arrangement. The stage movement can be controlled very accurately and the specimen may be fixed in a substrate. Also, the microscope is used in a controlled environment—for example mounted on a vibration isolation platform in a laboratory with a custom illumination setup so that the optical tolerances of the system (alignment and orientation of optical components and the stage) are very tight. Therefore, coarse alignment of the captured tiles for mosaicking is fairly accurate, the lighting even, and the transform between the tiles typically well represented by an affine transform—which includes the geometric operations of scaling, rotations, translations, and shear. On the other hand, the scale of certain important features of a specimen can be of the order of several pixels and the features can be densely arranged over the captured tile images. This means that the required stitching accuracy for virtual microscopy is very high. Additionally, given that the microscope can be loaded automatically and operated in batch mode, the processing requirements are also high.
The image registration process compares the pixels in the overlapping regions of two neighbouring images to determine the relative deformations in the images. In some systems all pixels in the overlapping regions in both images are used to calculate this deformation. However, the speed of the process can be significantly improved by only taking measurements at small image patches within the overlap regions. These patch-based techniques can be an order of magnitude faster and when the distortions present in the image are small—as is often the case in a microscope—they can be highly accurate.
An important problem when using patch-based techniques is where to locate the small patches. Locating patches in areas that contain a lot of texture is important to obtain an accurate estimate of the shift between corresponding patches in different images. There are several techniques to analyse the texture content of a patch and whether this texture is either 1D or 2D. However, it is not only important to place patches at locations with texture but in addition it is important to space them well throughout the image.
An existing technique used to choose well distributed patches is to select locations where the texture is a maximum within a disk of a fixed radius centered at that location. The problem with this technique is that the number of points that it selects depends upon the image content. An improvement to this technique has been introduced that effectively varies the radius of the disk to give a fixed number of points in all images. However, this technique is significantly slower than using a fixed radius. Other techniques have been proposed to improve the speed of the variable radius technique by comparing to an approximate radius which can be done much faster. These techniques give priority to locations that are local maxima of texture over a larger radius, and do not distinguish between low texture points versus high texture points with similar but unequal radii.
Alternative techniques that have been proposed include segmenting the image into a fixed grid of cells and choosing a fixed number of high texture locations from each cell. This does not work well if image features are localized to a small number of cells, and if many cells are blank this approach chooses many locations that are not useful for image registration Improvements to this technique have been proposed which adaptively subdivide the cells depending on the image content within them and which therefore choose more points where there is more texture. These techniques nonetheless are not able to reliably select patch locations that give good estimates of distortion between images when image noise produces spurious texture. Because of this a large number of patches need to be used to get a reliable estimate of the distortion. There is therefore a need for a method of selecting patch locations that overcomes the limitations of prior-art approaches.